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DESCRIPTION 



IMAGE SYSTEM 



5 Technical Field 

The present invention relates to an image 
system and an image identification system. 

Background Art 

10 Conventionally, an image sequence 

description system has been employed with intent to 
describe features used for retrieving a similar 
image sequence (a group of frames contained in a 
video) . An image sequence retrieving system using 

15 an image sequence description system is described 
in a first conventional art; A. Mufit Ferman et a 1 . , 
" GROUP- OF- FRAME/ PICTURE COLOR HI S TOG RAM 
DESCRIPTORS FOR MULTIMEDIA APPLICATIONS", 
(ICIP2000) . Fig. 1 shows the image sequence 

20 retrieving system disclosed in the first 
conventional art. 

With reference to Fig. 1, the image sequence 
retrieving system has a histogram generating unit 
11, a representative histogram generating unit 12, 

25 a description file generating unit 13/ a 

representative histogram storage unit 14, a 
histogram generating unit 15, a representative 
histogram generating unit 16, a similarity 
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calculating unit 17, a similarity sorting unit 18 
and a similar image displaying unit 19. 

The histogram generating unit 11 extracts a 
color histogram from each of frames of a target image 
5 sequence which is registered in a database. The 
representative histogram generating unit 12 
calculates an average value of each bin of the color 
histograms of all the frames which are extracted by 
the histogram generating unit 11. The description 

10 file generating unit 13 describes the calculated 
average value based on a predetermined format/ a n d 
then stores the described average value into the 
representative histogram storage unit 14. 

The histogram generating unit 15 extracts a 

15 color histogram from each of frames of an image 
sequence which is given as a query image. The 
representative histogram generating unit 16 
calculates as a representative histogram, an 
average value of each bin of the color histograms 

20 of all the frames which are extracted by the 
histogram generating unit 15. The similarity 
calculating unit 17 calculates a similarity between 
the representative histogram generated by the 
representative histogram generating unit 16 and 

25 each of the representative histograms stored in the 
representative histogram storage unit 14. The 
similarity sorting unit 18 rearranges the 
registered images in the order of higher similarity. 



The similar image displaying unit 19 displays the 
images in the order of the higher similarity. 

Another typical image identification system 
(second conventional art) is disclosed in Japanese 
5 Laid-open Patent Application No . 2 00 0-2 598 32. Fig. 
2 shows an image sequence retrieving system 
according to the second conventional art. With 
Reference to Fig. 2, the image sequence 
identification system has a feature extraction 

10 frame selecting unit 21, a color layout group 

generating unit 22, a description file generating 
unit 23, a color layout group storage unit 24, a 
feature frame selecting unit 25, a color layout 
group generating unit 26 and an image identification 

15 unit 27. 

The feature extraction frame selecting unit 
21 selects a set of target frames fromwhich features 
are to be extracted from a target image sequence 
which is registered in a database. The color layout 

20 group generating unit 22 extracts a feature 

indicative of a color layout feature of an image from 
each of the selected frames. The description file 
generating unit 23 describes the calculated feature 
group based upon a predetermined format, and then 

25 stores the described feature group into the color 
layout group storage unit 24. 

The feature extraction frame selecting unit 
25 selects a set of target frames fromwhich a feature 



is to be extracted from an image sequence which is 
given as a query image. The color layout group 
generating unit 26 extracts a feature indicative of 
a color layout feature of an image from each of the 
5 selected frames. The image identification unit 27 
sequentially compares a color layout group 
generated by the color layout group generating unit 
26 with a color layout group stored in. the 
representative color layout group 24 every frame. 

10 Then, the image identification unit 27 judges as to 
whether or not the query image sequence is 
consistent with any one of the image sequences 
stored in the color layout group storage unit. The 
image identification unit 27 outputs the judgment 

15 result as an identification result. 

However, in the a b o v e - d e s c r i b e d conventional 
systems, there are the following problems. 

In the system of the first conventional art, 
since only one color histogram representative of the 

20 image sequence is compared, a total number of the 
comparing operations is suppressed to a small number 
and the high-speed retrieving process operation can 
be realized. However, it is not possible to obtain 
a sufficient performance for identifying a image 

25 sequence. Also, since the layout structure is not 
considered in the color histogram, such an image 
which has the same color but different layout can 
be mistakenly retrieved. 



Also, in the systemof the second conventional 
art, since the feature is compared with each other 
in frame, a sufficiently high identification 
performance can be obtained. However, since the 
5 feature is compared in frame, a total number of the 
comparing operations is large and hence higher 
processing cost is required. 

In relation to the above description, 
Japanese Laid-open Patent Application No. Hei-7- 

10 79206 discloses an automatic CM identification 

system. The automatic CM identification system of 
the conventional art has CM extracting apparatuses 
provided for respective of TV broadcasting areas, 
and a CM identification apparatus communi ca tab 1 y 

15 connected via communication paths to the plurality 
of CM extracting apparatuses. The CM extracting 
apparatus has a plurality of CM scanners and a CM 
extraction control unit. Respective of the 
plurality of CM scanners are provided in respective 

20 of CM identification target stations within a 
predetermined TV broadcasting area. Each CM 
scanner receives a broadcasting program of one 
corresponding station and detects CM audio data. 
The CM extraction control unit transmits CM data via 

25 the communication path to the CM apparatus. The CM 
data include the CM audio data detected by the 
plurality of CM scanners, a name of the station which 
has broadcasted the CM, the time when the CM was 



broadcasted, and a time duration of the CM in seconds. 
The CM identification apparatus has a CM reference 
data storage unit, a CM record file, and a comparing 
unit. The CM reference data storage units are 
5 provided in correspondence with respective of the 
TV broadcasting areas. Each of the CM reference 
data storage units stores CM reference data with 
respect to every CM identification target station. 
The CM reference data include audio data for 

10 comparing CMs which are scheduled to be broadcasted 
and adjunctive data containing identification codes 
of the CMs which are scheduled to be broadcasted. 
The CM record files are provided in correspondence 
with respective of CM identification target 

15 stations within each TV broadcasting area. The 
comparing unit compares the CM audio data contained 
in CM data received from the CM extracting apparatus 
of each of the YV broadcasting areas with the 
c omp a r i n g - p u r p o s e audio data contained in the CM 

20 reference data of the corresponding station, which 
has been stored in the CM reference data storage unit 
for the corresponding TV broadcasting area. When 
the CM audio data is coincident with the 
compar ing-purpo s e audio data, both the adjunctive 

25 data contained in the CM reference data and the CM 
broadcasting time data contained in the CM data are 
recorded in the corresponding CM recording file. 

Also, Japanese Laid-open Patent Application 
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No. 2001-92486 discloses a high-speed signal 
retrieving apparatus. In this conventional art, a 
portion which resembles a previously registered 
reference signal is retrieved from an input signal. 
5 A reference feature series is derived from the 
reference signal. An input feature series is 
derived from the input signal, and an interest 
window is set with respect to this input feature 
series. A similarity between the reference feature 

10 series and the input feature series within the 
interest window is calculated. Based on the 
calculated similarity, a skip width of the interest 
window is calculated, and then the interest window 
is moved by the skip width. At the position of the 

15 moved interest window, a judgment is made on the 
basis of the calculated similarity as to whether or 
not the moved interest window must be returned in 
the time reverse direction. The similarity 
calculating step, the skip width calculating step 

20 and the back track necessity judging step are 
repeatedly carried out. Thus, similarities 
between the reference signal and the plural 
positions of the input signal are calculated, the 
similarities are compared with a preset target 

25 similarity, and then whether the reference signal 
is present at the relevant positions of the input 
signal or not is determined. 

Also, Japanese Laid-open Patent Application 
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No. 2001-101205 discloses a video structuring 
apparatus. The video structuring apparatus of this 
conventional art has a video recording apparatus, 
an image feature extracting apparatus for 
5 extracting a feature of an image, and a structure 
recording medium for recording the extracted 
feature in combination with a digital image. The 
recorded image is segmented by a cut/shot changing 
point recording apparatus. The information 

10 acquired by a representative frame extracting 

apparatus, a camera information recording apparatus 
and an environmental information recording 
apparatus are added to the segmented images by an 
information adding apparatus, and then the 

15 i n f o rma t i on- a dd e d segmented images are managed. 

Also, Japanese Laid-open Patent Application 
No. 2001-359125 discloses a CM investigation 
apparatus. The CM investigation apparatus of this 
conventional art is installed in a home, and data 

20 related to commercials which have been viewed in the 
home are transmitted as CM i n v e s t i g a t i o n- p u r p o s e 
data to a CM investigation center. The CM 
investigation apparatus has a receiving unit and a 
reception data storage unit. The receiving unit 

25 receives compressed CM data which is generated by 
compressing a data amount of an investigation target 
commercial video by using a specific algorithm among 
output data broadcasted from each of broadcasting 



stations. The reception data storage unit stores 
thereinto the received compressed CM data. To 
store output data, a compressed output data is 
recorded which is generated by compressing a data 
5 amount of broadcasting output data by using a 

specific algorithm. .The broad casting output data 
is a data which has been actually viewed in a 
broadcasting receiver. The compressed output data 
is compared with the compressed CM data so as to 

10 determine the viewed commercial. The 
determination is transmitted to the CM 
investigation center. 

Also, Japanese Laid-open Patent Application 
No. 2002-117037 discloses an image retrieving 

15 apparatus. The image retrieving apparatus of this 
conventional art retrieves an image which resembles 
a predetermined query image from a retrieval target 
moving picture. In this image retrieving apparatus, 
a frame feature extracting unit extracts a feature 

20 of at least a part of frames which are contained in 
the retrieval target moving picture. Then, the 
frame feature extracting unit outputs the extracted 
feature as a frame feature. A frame feature storage 
unit stores thereinto the frame feature outputted 

25 from the frame feature extracting unit. An image 
feature extracting unit extracts a feature of a 
query image, and outputs the extracted feature as 
an image feature. A similarity calculating unit 
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compares the frame feature stored in the frame 
feature storage unit with the image feature 
outputted from the image feature extracting unit so 
as to calculate a similarity between them. A frame 
5 feature synthesizing unit synthesizes such frame 
features among the frame features stored in the 
frame feature storage unit as has the similarities 
satisfying a predetermined condition into at least 
one group. A similar image selecting unit selects 

10 at least one frame feature having the highest 

similarity within the group generated by the frame 
feature synthesizing unit. An image having the 
frame feature selected by the similar image 
selecting unit is shown as a retrieval result. 

15 Also, Japanese Laid-open Patent Application 

No. 2002-335473 discloses a system for extracting 
retrieval information used for retrieving moving 
picture contents. In this conventional art, a 
moving picture content is subdivided every scene of 

20 the moving picture content. An image thumbnail 
unit selects a representative still image for every 
scene. Then, the representative still images of 
the respective scenes are stored in a moving picture 
retrieval information database in accordance with 

25 a storyline of the moving picture content. 

Also, National Publication of the. Translated 
Version of PCT Application No. 2002-537591 
discloses a description scheme of multimedia 
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archives. In this conventional art, a digital 
storage sub-system stores a multimedia record and 
its record description in accordance with a 
multimedia description scheme. A computer 
5 processing apparatus has at least one cluster which 
accesses the record description and relates at least 
two records within the digital storage sub-system 
based upon attribute of the record description. 
The computer processing apparatus generates an 

10 archive description which has an aggregate 

structure description scheme for assigning an index 
to the at least one cluster. The computer 
processing apparatus is connected to the digital 
storage sub-system under operable condition. An 

15 archive description storage apparatus is connected 
to the computer processing apparatus under operable 
condition in order to store archive description 
records . 

Japanese Patent No. 3065314 discloses a 
20 high-speed signal retrieving method. In this 

conventional art, a feature series is generated as 
to a reference signal which is a previously recorded 
acoustic signal. An input signal interest window 
having a constant time length is set with respect 
25 to an input signal which is an acoustic signal 

inputted. A feature series as to the input signal 
within the input signal interest window is generated. 
An input signal similarity value is calculated which 



indicates a similarity between the generated 
feature series and the generated feature series. 
Based on the calculated input signal similarity 
value, a movable amount indicative of an amount for 
5 which the input signal interest window can be moved 
is calculated. Based on the calculated movable 
amount, a position of the input signal interest 
window is determined, and then the input signal 
interest window is set to the determined position. 

10 Then, the above-mentioned operations are repeatedly 
carried out. Input signal similarity values are 
calculated as to respective positions of the input 
signal interest window. Each of the input signal 
similarity values is compared with a predetermined 

15 threshold value. On the basis of the comparison 
result, a judgement is made as to whether or not the 
reference signal is present at a current position 
on the input signal which is indicated by the input 
signal interest window. 

20 

Disclosure of Invention 

It is therefore an object of the present 
invention to provide an image system and an image 
description system, which are capable of speeding 
25 up an identification process for an image sequence 
without deteriorating identification precision by 
describing image sequences such that image 
sequences which require identification in a unit of 



frame can be selected. 

In an aspect of the present invention, an 
image description system has: a feature extracting 
unit for extracting layout features from respective 
5 of frames of an image sequence; and a representative 
feature calculating unit for calculating a layout 
feature representative of the image sequence from 
a group of the layout features extracted by the 
feature extracting unit. 

10 In another aspect of the present invention, 

an image description system has: a feature 
extracting unit for extracting layout features from 
respective of frames of an image sequence; a 
representative feature calculating unit for 

15 calculating a layout feature representative of the 
image sequence from a group of the layout features 
extracted by the feature extracting unit; and a 
representative layout feature storage unit for 
storing the representative layout feature 

20 calculated by the representative feature 
calculating unit. 

In still another aspect of the present 
invention, an image description system has: a 
feature extracting unit for extracting layout 

25 features from respective of frames of an image 

sequence; a representative feature calculating unit 
for calculating a layout feature representative of 
the image sequence from a group of the layout 



features extracted by the feature extracting unit; 
a representative layout feature storage unit for 
storing the representative layout feature 
calculated by the representative feature 
5 calculating unit; and a layout feature group storage 
unit for storing the group of the layout features 
calculated by the feature extracting unit. 

In the above-description, it is preferable 
that the representative feature calculating unit 

10 calculates average values of respective elements of 
the layout feature extracted by the feature 
extracting unit, and then sets the average values 
as a representative feature. 

It is also preferable that the representative 

15 feature calculating unit rearranges values of 

respective elements of the layout feature extracted 
by the feature extracting unit in either ascending 
order or descending order, calculates a median and 
then sets the calculated median as a representative 

20 feature. 

The layout feature can be color layout 
information, or can be edge histogram information. 

Also, the representative feature may be 
preferably described based on a description scheme. 

25 In the description scheme, the representative 

feature is selectable from a plurality of features 
including at least "ColorLayout" and 
"EdgeHistogram" , and a representative feature 



calculating method is selectable from a plurality 
of methods including at least "Average" , "Median" 
and "Intersection". 

In still another aspect of the present 
5 invention, an image description software product 
executable on a computer realizes: a feature 
extracting function for extracting layout features 
from respective of frames of an image sequence; and 
a representative feature calculating function for 

10 calculating a layout feature representative of the 
image sequence from a group of the layout features 
extracted by the feature extracting function. 

In still another aspect of the present 
invention, an image description method includes: a 

15 feature extracting step for extracting layout 
features from respective of frames of an image 
sequence; and a representative feature calculating 
step for calculating a layout feature 
representative of the image sequence from a group 

20 of the layout features extracted in the feature 
extracting step. 

In still another aspect of the present 
invention, an image identification system has: a 
representative layout feature storage unit for 

25 storing a layout feature representative of an image 
sequence as a representative layout feature; and an 
image sequence selecting unit for selecting an image 
sequence which resembles a query image sequence by 

15 



comparing the representative layout feature of the 
query image sequence stored in the representative 
layout feature storage unit with the representative 
layout feature of a registered image sequence stored 
5 in the representative layout feature storage unit. 

In still another aspect of the present 
invention, an image identification system has: a 
representative layout feature storage unit for 
storing a layout feature representative of an image 

10 sequence as a representative layout feature; a 
feature extracting unit for extracting layout 
features from respective of frames of a query image 
sequence; a representative feature calculating unit 
for calculating a layout feature representative of 

15 the query image sequence from a group of the layout 
features extracted by the feature extracting unit; 
and an image sequence selecting unit for selecting 
a sequence which resembles the query image sequence 
by comparing the representative layout feature 

20 calculated by the representative feature 

calculating unit with the representative layout 
feature stored in the representative layout feature 
storage unit. 

In still another aspect of the present 

25 invention, an image identification system has: a 
representative layout feature storage unit for 
storing a layout feature representative of an image 
sequence as a representative layout feature; a 

16 

\ 



layout feature group storage unit for storing layout 
features associated with respective of frames of an 
image sequence; a feature extracting unit for 
extracting layout features from respective of 
5 frames of a query image sequence; a representative 
feature calculating unit for calculating a layout 
feature representative of the query image sequence 
from a group of the layout features extracted by the 
feature extracting unit; an image sequence 

10 selecting unit for selecting a sequence which 

resembles the query image sequence by comparing the 
representative layout feature calculated by the 
representative feature calculating unit with the 
representative layout feature stored in the 

15 representative layout feature storage unit; and an 
identification unit for matching the group of layout 
features extracted by the layout feature extracting 
unit against the layout features stored in the 
layout feature group unit as to the image sequence 

20 selected by the image sequence selecting unit. 

In the above-description, it is preferable 
that the representative feature calculating unit 
calculates average values of respective elements of 
the layout feature extracted by the feature 

25 extracting unit, and then sets the average values 
as a representative feature. 

It is also preferable that the representative 
feature calculating unit rearranges values of 



respective elements of the layout feature extracted 
by the feature extracting unit in either ascending 
order or descending order, calculates a median and 
then sets the calculated median as a representative 
5 f ea tur e . 

The layout feature can be color layout 
information, or can be edge histogram information. 

Also, the representative feature may be 
preferably described based on a description scheme. 

10 In the description scheme, the representative 

feature is selectable from a plurality of features 
including at least " ColorLayout" and 
"EdgeHistogram" , and a representative feature 
calculating method is selectable from a plurality 

15 of methods including at least "Average", "Median" 
and "Intersection". 

In still another aspect of the present 
invention, an image identification software product 
executable on a computer realizes: a storage 

20 function for storing a layout feature 

representative of an image sequence as a 
representative layout feature in a storage function 
of the computer; and a selection function for 
selecting an image sequence which resembles a query 

25 image sequence by comparing a representative layout 
feature of the query image sequence stored by the 
storage function with a representative layout 
feature of a registered image sequence. 



In still another aspect of the present 
invention, an image identification method includes: 
a storing step for storing a layout feature 
representative of an image sequence as a 
5 representative layout feature; and an image 

sequence selecting step for selecting an image 
sequence which resembles a query image sequence by 
comparing a representative layout feature of the 
query image sequence stored in the storing step with 

10 a representative layout feature of a registered 
image sequence. 

In still another aspect of the present 
invention, a description scheme is for describing 
a feature of an image sequence, in which a 

15 representative feature is selectable from a 
plurality of features including at least 
"ColorLayout" and "EdgeHistogram" , and a 
representative feature calculating method is 
selectable from a plurality of methods including at 

20 least "Average", "Median" and "Intersection". 

According to the present invention, as 
described above, it is possible to skip all 
operations of comparing the layout feature groups 
in a unit of frame, which require high process cost, 

25 with respect to registered image sequences which are 
determined to have no possibility of having the same 
contents as a query image sequence at the time when 
comparison between the representative layout 



features is carried out. Thus, it is possible to 
reduce cost of the identification process largely 
without deteriorating precision of the 
identification . 
5 Also, representative layout features of 

image sequences are previously generated and stored. 
Thus, it is possible to provide in low cost an 
information used for selecting registered image 
sequences which have the possibility of having the 

10 same content as the query image sequence. 

Furthermore, the representative layout 
features are described on the basis of a common 
description scheme. It is therefore possible to 
exchange the representative layout features between 

15 a plurality of systems, and to read out the 

representative layout features from a plurality of 
storage devices. 

Brief Description of Drawings 

20 Fig. 1 is a diagram showing a system of the 

first conventional art; 

Fig. 2 is a diagram showing a system of the 
second conventional art; 

Fig. 3 is a block diagram showing a 
25 configuration according to a first embodiment of the 
present invention; 

Fig. 4 is a flow chart showing a procedure of 
generating a representative layout feature by means 

20 



of average in the first embodiment; 

Fig. 5 is a flow chart showing a procedure of 
generating a representative layout feature by means 
of median acquisition in the first embodiment; 
5 Fig. 6 is a diagram showing an example of color 

layout values by way of an averaging operation in 
the first embodiment; 

Fig. 7 is a diagram showing an example of color 
layout values by acquiring median in the first 
10 embodiment; 

Fig. 8 is a diagram showing a description, 
scheme in the first embodiment; 

Fig. 9 is a flow chart showing a procedure of 
identifying an image according to the first 
15 embodiment; 

Fig. 10 is a block diagram showing a 
configuration according to a second embodiment of 
the present invention; 

Fig. 11 is a block diagram showing a 
20 configuration according to a third embodiment of the 
present invention; 

Fig. 12 is a block diagram showing a 
configuration according to a fourth embodiment of 
the present invention; 
25 Fig. 13 is a block diagram showing a 

configuration according to a fifth embodiment of the 
present invention; 

Fig. 14 is a diagram showing an aggregation 

21 



method which can be used as to respective features 
in the first embodiment; 

Fig. 15 is a diagram showing a binary 
description syntax in the first embodiment; 
5 Fig. 16 is a diagram showing an example of a 

binary expressed description of the aggregation 
method in the first embodiment; and 

Fig. 17 is a diagram showing another 
description scheme in the first embodiment. 
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Best Mode for Carrying Out the Invention 

An image identification system according to 
the present invention will be described in detail 
with reference to the accompanying drawings. 
15 (FIRST EMBODIMENT) 

Fig. 3 is a block diagram showing an example 
of a configuration of an image identification system 
according to a first embodiment of the present 
invention . 

20 As shown in Fig. 3, the ima.ge identification 

system according to the present embodiment has a 
description unit 31, a storage unit 32 and an 
identification unit 33. The description unit 31 
describes a signal feature of an image by extracting 

25 a feature from an image sequence. The feature 

storage unit 32 stores a feature of an image to be 
registered. The identification unit 33 identifies 
a query image sequence. 



The description unit 31 has a layout feature 
extracting unit 101 and a representative layout 
feature calculating unit 102. 

The layout feature extracting unit 101 
5 extracts a layout feature from each of frames of a 
registration image sequence to be registered. It 
is difficult according to a color histogram to 
identify an image the color of which is different 
from the color of a query image, but the content of 

10 which is identical to the content of the query image. 
Among images having the same contents, even when 
colors are shifted from each other, compositions of 
images are not substantially changed. Therefore, 
in order to identify an image, it is considered to 

15 be effective to employ a layout feature which 

represents rough composition of an image. Thus, 
the layout feature is utilized. Layout features of 
various kinds may be available. These layout 
features are defined in the International Standard 

20 ISO/IEC 15838-3 MPEG-7 VISUAL. Any of "COLOR 
LAYOUT DESCRIPTOR" for representing a layout 
structure of a color and " EDGEH I S TOG RAM DESCRIPTOR" 
for representing a layout structure of an edge will 
be used. 

25 The representative layout feature 

calculating unit 102 calculates a representative 
layout feature representative of the registration 
image sequence from a group of layout features which 

23 
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are extracted by the layout feature extracting unit 
101. Although various calculating methods may be 
conceived in calculating the representative layout 
feature, averaging the group of layout features and 
5 utilizing a median will be employed. Figs. 4 and 

5 show flow charts for these calculating methods, 
respect i vel y . 

Fig. 4 is a flow chart showing a procedure of 
generating the representative layout feature by 
10 means of averaging. The procedure is shown by 
exemplifying the "COLOR LAYOUT" as the layout 
feature, which has 6 parameters of luminance and 3 
parameters of color difference as features. Fig. 

6 shows values of respective elements of the "COLOR 
15 LAYOUT" for each frame. An element "YDC" denotes 

a DC component of the luminance; elements "CbDC" and 
"CrDC" denote DC components of the color difference; 
an element " Y AC [ n ] " denotes the n-th AC component 
of the luminance; and elements "CbAC [n] " and 
20 "CrAC[n]" denote the n-th AC components of the color 
difference. 

First, a total value of each element of the 
layout feature is reset to zero (Step Al). Next, 
a layout feature of a frame extracted by the layout 
25 feature extracting unit 101 is read every frame. 
The values of each element of the read frame features 
are added to each other as to all of the frames (Steps 
A2 to A4 ) . For instance, when an attention is paid 



to the element " Y DC (DC component of luminance)" in 
the table shown in Fig. 6, a total value amounts to 
3 1+11+15+21+32=110. The total value for each 
element of the layout features of all the frames is 
5 normalized by the number of the frames, and thus an 
average value is calculated (Step A5) . When an 
attention is paid to the element " YDC (DC component 
of luminance)" in the table shown in Fig. 6, an 
average value becomes 110/5=22. A set of the 

10 calculated average values is the representative 
layout feature (Step A6) . In the example of the 
table shown in Fig. 6, the values in an AVERAGE record 
correspond to the representative layout feature. 

Fig. 5 is a flow chart showing a procedure of 

15 generating the representative layout feature by 
means of median. The procedure is shown by 
exemplifying the "COLOR LAYOUT" as the layout 
feature, which has 6 parameters of luminance and 3 
parameters of color difference as features. Fig. 

20 5 shows values of respective elements of the "COLOR 
LAYOUT" for each frame. An element "YDC" denotes 
a DC component of the luminance; elements "CbDC" and 
"CrDC" denote DC components of the color difference; 
an element " Y AC [ n ] " denotes the n-th AC component 

25 of the luminance; and elements "CbAC [n] " and 

"CrAC [n] " denote the n-th AC components of the color 
difference . 

First, a total value of each element of the 

25 



layout feature is reset to zero (Step Bl) . Next, 
layout features of all the frames extracted by the 
layout feature extracting unit is read. The values 
of the elements of all of the read frame features 
5 are rearranged every element in ascending order 
(Steps B2 to B3) . For instance, when an attention 
is paid to the element "YDC (DC component of 
luminance) " in the table shown in Fig. 7, rearranged 
result is given as follows: 32, 31, 21, 15, and 11. 

10 A median value is acquired from a string of numerals 
arranged in ascending order for each element of 
layout features of the all frames, and then a median 
is determined (Step B4) . When an attention is paid 
to the element YDC (DC component of luminance) in 

15 the table shown in Fig. 7, the median is 21. A set 
of the calculated medians is the representative 
layout feature (Step B5) . In the example of the 
table shown in Fig. 7, the values in a MEDIAN record 
correspond to the representative layout feature. 

20 In addition, another method may be conceived 

in which, after each element has been rearranged in 
ascending order, only a minimum value is selected 
for each value instead of determining the median. 
The extracted representative feature is 

25 described on the basis of a predetermined 

description scheme. Since the description scheme 
is previously defined and is shared among terminals, 
it is possible to exchange the description data with 



other terminals and utilize the description data 
among a plurality of terminals in conjunction 
there to . 

Various schemes may be employed as the 
5 description scheme. Fig. 8 shown an example of the 
description scheme. The description scheme is 
defined such that any one of "ColorLayout" and 
"EdgeHistogram" is selected as a representative 
layout feature, and any one of averaging (Average) , 

10 a median (Median) and a minimum value (Intersection) 
is selected as a method of generating the 
representative layout feature (aggregation) . 

Also, it is possible to describe a feature in 
a binary expression instead of describing it based 

15 on the description scheme. Fig. 15 shows an example 
of syntax of a feature description in the binary 
expression. On describing a feature in the binary 
expression, whether a method of generating a 
representative layout feature (aggregation) is 

20 defined or not is first described. In the case when 
an aggregation is described, "1" is allocated, 
whereas in the case when an aggregation is not 
described, "0" is allocated. Next, in the case when 
the aggregation is described, a method of generating 

25 a representative layout feature is described. In 
the case of the average (Average), "001" is 
allocated; in the case of the median (Median) , "010" 
is allocated; and in the case of the minimum value 



(Intersection), "Oil" is allocated. Then, an 
identifier is a 1 1 o c a t e d wh i c h indicates the feature 
selected as the representative layout feature. In 
the case of ColorLayout, "4" is allocated, whereas 
5 in the case of EdgeHistogram, "8" is allocated. 
Further, a content of the selected feature in the 
binary expression follows. Such a binary 
expression method defined by ISO/IEC 15938-3 is 
employed as the binary expression method. 
10 Semantics in the description scheme shown in 

Fig. 8 and the binary syntax shown in Fig. 15 is given 
as follows: 

• Descriptor ID 

This field corresponds to an identifier for 
15 specifying a type of a representative feature. In 
the case of ColorLayout, 4 is allocated. In the case 
of EdgeHi s togram, 8 is allocated. The other values 
are prohibited. 

• Descriptor 

20 This field indicates a feature generated by 

using description tools which are defined in the 
International Standard ISO/IEC 15838-3. Available 
tools are ColorLayout and E d g e H i s t o g r am . 

• AggregationFlag 

25 This field indicates as to whether an 

aggregation is defined or not. In the case when "1" 
is allocated, an attribute of the aggregation is 
given as follows: 



aggregation 
In this optional field, an aggregation method 
for generating a representative feature from a group 
of f r ame s / ima ge s is designated. The aggregation is 
5 carried out for the entire elements contained in the 
group of f r ame s / image s . Any one of the following 
types can be applied. 

• Average 

An aggregation by an average imp lies that each 
10 feature element of each frame/image is sequentially 
added, the total value is normalized by the number 
of the frames/images, and thus each element of a 
feature is calculated. 

• Median 

15 An aggregation by a median implies that a list 

is produced in which values of each feature element 
of the f r ame s / ima g e s are rearranged in ascending 
order, and a median is obtained from this list so 
as to be used as a representative value. 

20 - Intersection 

An aggregation by an intersection implies 
that each element of a representative feature is 
obtained by calculating a minimum value as to each 
element from features which are contained in all of 

25 the f r ame s / image s . 

It should be noted that a use of a part of the 
aggregation methods is prohibited. The 
aggregation methods which can be used for each 
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feature are designated by a mark " Y " in Fig. 14. 

It shows that the Average and the Median can 
be applied as the aggregation method to both of the 
ColorLayout and the EdgeHistogram. 
5 When describing the aggregation method 

( Aggregat ionType ) in a binary expression, a table 
shown in Fig. 16 is used. More specifically, in the 
case of the average (Average), "001" is allocated; 
in the case of the median (Median), "010" is 

10 allocated; and in the case of the minimum value 
(Intersection), "011" is allocated. 

Fig. 17 shows an example of another 
description scheme. As the representative feature, 
in addition to the "Colorlayout" and the 

15 "EdgeHistogram", one or a plurality of all image 
features defined in MPEG-7 can be instanced. It 
should be noted that "Picturetype" is a decription 
scheme which aggregates still image features. 

Also, the defined aggregation method is not 

20 necessary selected, but an undefined (unspecified) 
one can be selected as the aggregation method. By 
leaving an aggregation method unspecified, it 
becomes possible to define and utilize an 
aggregation method which is specific to a system. 

25 As explained above, by describing the 

representative layout feature on the basis of a 
common description scheme, it is possible to 
exchange the representative layout features among 



a plurality of systems and to read out the 
representative layout features from a plurality of 
storage devices. 

Moreover, by describing the representative 
5 layout feature in the binary expression based on the 
defined binary expression syntax, it is possible to 
describe a feature with considerably small amount 
of information, as compared with the case when 
describing the representative layout feature on the 
10 basis of the description scheme. 

The storage unit 32 has a representative 
layout feature storage unit 103 and a layout feature 
group storage unit 104. 

The representative layout feature storage 
15 unit 103 stores the representative layout feature 
calculated by the representative layout feature 
calculating unit 102. Also, the layout feature 
group storage unit stores layout features of 
respective of the frames which are extracted by the 
20 layout feature extracting unit 101. 

The identification unit 33 has a layout 
feature extracting unit 105, a representative 
layout feature calculating unit 106, an image 
sequence selecting unit 107 and an image sequence 
25 identification unit 108. 

The layout feature extracting unit 105 
extracts a layout feature from each of frames of an 
inputted query image sequence. The extraction 



method is similar to that of the layout feature 
calculating unit 101. 

The representative layout feature 
calculating unit 106 calculates a representative 
5 layout feature representative of the query image 
sequence from a group of the layout features 
extracted by the layout feature extracting unit 105. 
The calculation method is similar to that of the 
representative layout feature c a 1 c u 1 a t i ng un i t 102. 

10 The image sequence selecting unit 107 

evaluates a similarity by comparing the 
representative layout feature of the query image 
sequence generated by the representative layout 
feature calculating unit 106 with representative 

15 layout features of the registered image sequences 
stored in the representative layout feature storage 
unit 103. Then, the image sequence selecting unit 
107 selects only registered image sequences which 
are judged to have possibility of having the same 

20 content as the query image sequence. 

The image sequence identification unit 108 
compares the group of layout features of the query 
image sequence generated by the layout feature 
calculating unit 105 with the group of layout 

25 features of the registered image sequence stored in 
the layout feature group storage unit 104, as to the 
registered image sequences selected by the image 
sequence selecting unit 107. The image sequence 



identification unit 108 evaluates a similarity 
based on the comparison, and judges as to whether 
or not the content of the registered image sequence 
is identical to that of the query image sequence. 
5 When there is a registered image having the same 
content, the image sequence identification unit 108 
outputs a unique ID of the registered image sequence 
having the same content. 

A procedure of identifying an image sequence 

10 will be described with reference to Figs. 3 and 9. 
Fig. 9 is a flow chart showing the procedure of 
identifying an image sequence according to the 
present embodiment. First, representative layout 
features of all registration image sequences are 

15 stored beforehand in the representative layout 

feature storage unit 103 in combination with unique 
IDs indicative of respective images. Also, layout 
feature groups of all the registration image 
sequences are stored beforehand in the layout 

20 feature storage unit 104 in combination with the 
unique IDs indicative of respective images (Step 
CI ) . 

A layout feature and a representative layout 
feature are extracted from an inputted query image 
25 sequence by the layout feature extracting unit 105 
and the representative layout feature calculating 
unit 106, respectively (Steps C2 to C3) . 

Subsequently, the image sequence selecting 
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unit 107 reads one of the representative layout 
features of the registered image sequences, and then 
compares the read representative layout feature 
with the representative layout feature of the query 
5 image sequence (Steps C4 to C5) . The image sequence 
selecting unit 107 calculates a distance between the 
representative layout feature of the query image 
sequence and the representative layout feature of 
the one registered image sequence which has been 

10 read. Then, the image sequence selecting unit 107 
judges whether or not the calculated distance is 
smaller than or equal to a predetermined threshold 
value (Step C6). In the case when the calculated 
distance is smaller than or equal to the 

15 predetermined threshold value, the image selecting 
unit 107 judges that there is a possibility that the 
query image sequence has the same content as the one 
registered image sequence, and then outputs a unique 
ID of the relevant registered image sequence ("YES" 

20 in Step C6) . 

The image sequence identification unit 108 
reads a layout feature group corresponding to the 
unique ID of the image sequence which is judged to 
have the possibility of having the same content 

25 (Step C7) . The image sequence identification unit 
10 8 comp ares the read layout feature group with a 
layout feature group of the query feature group 
(Step C8 ) . As a method of c omp aring time-series 



data of the layout features, for example, the method 
described in Japanese Laid-open Patent Application 
No. 2000-259832 (the second prior art) can be 
employed, which enables the high-speed comparison. 
5 In the case when the registered image sequence is 
determined to have the same content as the query 
image sequence as a result of the comparison, the 
unique ID of the registered image sequence is 
outputted (Steps C9 to CIO). 

10 According to the first embodiment, as described 

above, the image sequence selecting unit 107 and the 
image sequence identification unit 108 are provided. 
It is therefore possible to skip all operations of 
comparing the layout feature groups in a unit of 

15 frame, which require high process cost, with respect 
to the registered image sequences which are 
determined to have no possibility of having the same 
contents as the query image sequence at the time when 
the comparison between the representative layout 

20 features is carried out. Thus, it is possible to 
reduce cost of the identification process largely 
without deteriorating precision of the 
identification. 

Also, the representative layout features of 

25 the image sequences are generated by the 

representative layout feature calculating unit 102, 
and are stored in the representative layout feature 
storage unit 104. Thus, it is possible to select 



with low cost the registered image sequences which 
have the possibility of having the same content as 
the query image sequence. 

(SECOND EMBODIMENT ) 
5 As shown in Fig. 10, an image identification 

system according to a second embodiment of the 
present invention does not have the layout feature 
extracting unit 105 and the representative layout 
feature calculating unit 106. Instead, a 

10 representative layout feature and a layout feature 
group of a query image sequence are previously 
extracted and stored in the representative layout 
feature storage unit 103 and the layout feature 
group storage unit 104, respectively. 

15 The image sequence selecting unit 107 

evaluates a similarity by comparing the 
representative layout feature of the query image 
sequence with the representative layout features of 
the registered image sequences stored in the 

20 representative layout feature storage unit 103. 
Then, the image sequence selecting unit 107 selects 
only registered image sequences which are judged to 
have possibility of having the same content as the 
query image sequence. 

25 The image sequence identification unit 108 

evaluates a similarity by comparing the layout 
feature group of the query image sequence with the 
layout feature group of the registered image 



sequence stored in the layout feature group storage 
unit 104, as to the registered image sequence 
selected by the image sequence selecting unit 107. 
Then, the image sequence identification unit 108 
5 judges whether or not the content of the selected 
registered image sequence is identical to the 
content of the query image sequence. In the case 
when there is a registered image having the same 
content, the image sequence identification unit 108 
10 outputs a unique ID of the registered image sequence. 
Since the operation of calculating features is 
eliminated, a higher speed processing becomes 
possible . 

According to the second embodiment, the image 
15 sequence selecting unit 107 and the image sequence 
identification unit 108 are provided. It is 
therefore possible to skip all operations of 
comparing the layout feature groups in a unit of 
frame, which require high process cost, with respect 
20 to the registered image sequences which are 

determined to have no possibility of having the same 
contents as the query image sequence at the time when 
the comparison between the representative layout 
features is carried out. Thus, it is possible to 
25 reduce cost of the identification process largely 
without deteriorating precision of the 
identification. 

(THIRD EMBODIMENT) 
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Fig. 11 shows an image identification system 
according to a third embodiment of the present 
invention. As shown in Fig. 11, the image 
identification system according to the third 
5 embodiment has a description unit 31, a storage unit 
35 and a selection unit 36. The description unit 
31 extracts a feature from an image sequence and 
describes the signal feature of the image. The 
storage unit 35 stores features of images to be 
10 registered. The selection unit 36 is provided for 
a query image sequence. 

The description unit 31 is similar to that in 
the first embodiment. 

The storage unit 35 has a representative 
15 layout feature storage unit 103. The 

representative layout feature storage unit 103 
stores a representative layout feature generated by 
the description unit 31. 

The selection unit 36 has a layout feature 
20 extracting unit 105, a representative layout 

feature calculating unit 106 and an image sequence 
selecting unit 107. Functions of the layout 
feature extracting unit 105, the representative 
layout feature calculating unit 106 and the image 
25 sequence selecting unit 107 are similar to those in 
the first embodiment. 

According to the third embodiment, the image 
sequence selecting unit 107 is provided. It is 



therefore possible to select registered image 
sequences which have no possibility of having the 
same content. Also, the representative layout 
features of the image sequences are generated by the 
5 representative layout feature calculating unit 102, 
and are stored in the representative layout feature 
storage unit 104. Thus, it is possible to select 
with low cost the registered image sequences which 
have possibility of having the same content. 

10 (FOURTH EMBODIMENT) 

Fig. 12 shows an image identification system 
according to a fourth embodiment of the present 
invention. As shown in Fig. 12, the image 
identification system does not have the layout 

15 feature extracting unit 105 and the representative 
layout feature calculating unit 106 which are 
provided in the third embodiment. Instead, a 
representative layout feature of a query image 
sequence is previously calculated and stored in the 

20 representative layout feature storage unit 103. 

According to the fourth embodiment, the image 
sequence selecting unit 107 is provided. It is 
therefore possible to select registered image 
sequences which have no possibility of having the 

25 same content. Also, the representative layout 

features of the image sequences are generated by the 
representative layout feature calculating unit 102, 
and are stored in the representative layout feature 



storage unit 104. Thus, it is possible to select 
with low cost the registered image sequences which 
have possibility of having the same content. 
(FIFTH EMBODIMENT) 
5 Fig. 13 shows an image identification system 

according to a fifth embodiment of the present 
invention. As shown in Fig 13, the image 
identification system according to the present 
embodiment has a description unit 31 and a storage 

10 unit 35. The description unit 31 extracts a feature 
from an image sequence and describes the signal 
feature of the image. The storage unit 35 stores 
features of images to be registered. The 
operations of description unit 31 and the storage 

15 unit 3.5 are similar to those is the third embodiment. 
According to the fifth embodiment, the 
representative layout features of the image 
sequences are generated by the representative 
layout feature calculating unit 102, and are stored 

20 in the representative layout feature storage unit 
104. Thus, it is possible to provide information 
used for selecting with low cost the registered 
image sequences which have possibility of having the 
same content . 

25 Next, a concrete example will be explained. 

A commercial identification system will be 
shown, in which a commercial (CM) is used as an image. 
In this system, a color layout descriptor is 
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generated based on a received broadcasting image, 
matching against each of CMs registered in a 
database is sequentially carried out, and then the 
CM on the air is identified. An identification 
5 result is notified to a user as a broadcasting 
investigation report. 

First, CM images with the duration of 15 
seconds are registered. A color layout is 
extracted from each of frames of a CM image. Then 

10 a color layout representative of each of the CM 
images is calculated from a group of the extracted 
color layouts. The calculated representative 
color layouts and the extracted color layout groups 
are beforehand stored in the CM database. 

15 Next, matching of a color layout arrangement 

of an image acquired from broadcast waves against 
a color layout arrangement of each of the CM images 
previously registered is carried out. Frames which 
are included within 15 seconds are sequentially 

20 inputted as query image sequences from the broadcast 
images. Color layouts are extracted from the query 
image sequence, and then a representative color 
layout is calculated from a group of the extracted 
color layouts. 

25 By the comparison between the representative 

color layouts, CMs having possibility of having the 
same content as the image on the air which 
corresponds to the query image sequence are 



selected. 

After that, comparison of the color layouts 
of all frames between the selected CMs and the query 
image sequence is made. The distances between the 
5 query image and all the selected CMs are calculated 
and a retrieval is made. In the case when a feature 
distance with respect to a CM which is judged as the 
most resembling CM is smaller than or equal to a 
predetermined threshold value, the video scene on 

10 the air is determined to be the CM. In order to 
realize a real time detection, it is required to 
complete the retrieving operation within a sampling 
interval of input images. An identification result 
is notified to a user as a broadcasting 

15 investigation report. 

According to the present embodiment, a 
comparison of the number of identification 
operations is made between a case that the comparing 
step of the representative color layout is included 

20 and a case that the comparing step of the 

representative color layout is not included and 
matching of color layouts of all frames contained 
in the query image sequence is made against all the 
CMs. The number of the identification operations 

25 can be reduced to approximately 10 % without 

deteriorating precision of the identification. 

It should be noted that the ab o v e - d e s c r ib e d 
embodiments are preferred examples of the present 
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invention. The present invention is not limited to 
these embodiment and can be variously modified 
without departing from the scope of the present 
invent ion. 

5 According .to the present invention, as 

described above in detail, it is possible to skip 
all operations of comparing the layout feature 
groups in a unit of frame, which require high process 
cost, with respect to the registered image sequences 

10 which are determined to have no possibility of 

having the same contents as a query image sequence 
at the time when comparison between the 
representative layout features is carried out. 
Thus, it is possible to reduce cost of the 

15 identification process largely without 

deteriorating precision of the identification. 

Also, representative layout features of 
image sequences are previously generated and stored. 
Thus, it is possible to provide in low cost an 

20 information used for selecting registered image 
sequences which have the possibility of having the 
same content as the query image sequence. 

Furthermore, the representative layout 
features are described on the basis of a common 

25 description scheme. It is therefore possible to 
exchange the representative layout features between 
a plurality of systems, and to read out the 
representative layout features from a plurality of 



storage devices. 

It should also be noted that the units for 
achieving various kinds of operations in the above 
embodiments can be realized in the form of hardware, 
or can be alternatively realized in the form of 
software. In this alternative case, the functions 
of the respective units can be realized by a CPU (not 
shown) in a general computer system executing the 
software which is stored in a memory (not shown) . 
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